Genotypic versus Behavioural Diversity for Teams of Programs under the 4-v-3 Keepaway Soccer Task

Keepaway soccer is a challenging robot control task that has been widely used as a benchmark for evaluating multi-agent learning systems. The majority of research in this domain has been from the perspective of reinforcement learning (function approximation) and neuroevolution. One of the challenges under multi-agent tasks such as keepaway is to formulate effective mechanisms for diversity maintenance. Indeed the best results to date on this task utilize some form of neuroevolution with genotypic diversity. In this work, a symbiotic framework for evolving teams of programs is utilized with both genotypic and behavioural forms of diversity maintenance considered. Specific contributions of this work include a simple scheme for characterizing genotypic diversity under teams of programs and its comparison to behavioural formulations for diversity under the keepaway soccer task. Unlike previous research concerning diversity maintenance in genetic programming (GP), we are explicitly interested in solutions taking the form of teams of programs. Introduction Symbiotic Bid-Based GP (SBB) is a hierarchical framework for symbiotically coevoling teams of simple programs over two distinct cycles of evolution (Kelly, Lichodzijewski, and Heywood 2012). The first cycle produces a library of diverse, specialist teams with limited capability. The second cycle builds more general and robust policies by re-using the library, essentially building generalist teams from multiple specialists. Thus, diversity maintenance is critical during the first stage of evolution to ensure the identification of a wide range of specialist behaviours. Keepaway soccer is a challenging benchmark task for multi-agent learning in which a team of K keepers must maintain possession of the ball while an opposing team of K 1 takers attempt to gain possession (Stone et al. 2006). The keepers must learn a policy that maximizes the length of play against the takers, which follow a pre-specified behaviour. The players’ sensors and actuators are noisy, making the task partially observable and highly stochastic. The size of the playing region and number of keepers vs. takers may be adjusted to scale the difficulty of the task. In Copyright c 2014, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. this work we are working with the 4-v-3 task configuration on a 25m x 25m field. The keepaway task has been dominated by value function-based approaches to reinforcement learning (Kalyanakrishnan and Stone 2009) and neuroevolution (Metzen et al. 2007). In the case of neuroevolution, genotypic diversity maintenance was shown to be a critical factor in the algorithm’s success. However, the maintenance of behavioural diversity is increasingly found to be more effective than diversity mechanisms operating solely in genotype space, in particular when the domain is deceptive, for example, due to a noisy fitness function (Gomez 2009). In this work we introduce novel methods for measuring genotypic and behavioural diversity among teams of programs, and empirically compare their efficacy under the keepaway task. Diversity Mechanisms In the keepaway task, the fitness of a team is the mean episode length over all games played. In order to promote population diversity, each team’s novelty must also factor into the selection process, where novelty refers to the mean genotypic or behavioural distance between a team and all other members of the same population. Several methods to balance fitness and novelty are possible, including fitness sharing, crowding, or multi-objective optimization. In this work we adopt a simple linear combination of fitness and novelty (Cuccu and Gomez 2011), thus each team’s score is defined prior to selection: score(tm